PHP MySQL - Import million of CSV records into table in short time Using LOAD DATA

14-04-2017

PHP MySQL - Import million of CSV records into table in short time Using LOAD DATA

Using LOAD DATA sql query, we can import large csv file into database table


I've always been trouble with uploading large CSV files into SQL databases. When we upload a CSV file, what we do mostly is, read each CSV records one by one, and we run an insert query inside a while loop. this is ok if the CSV records are in thousands. When we upload large CSV files using open and reading each record, this takes a huge load time. The solution is, we can use SQL LOAD DATA query. this will direct upload CSV records into the table.

Before starting, here is some configuration that needs to be done in php.ini

Edit post_max_size and upload_max_filesize in php.ini , it should be greater than size of CSV file being upload.
Uncomment mysqli.allow_local_infile = On in php.ini, So that, we can read data from upload CSV File.
That's all configuration we need to do. Next, I'm gonna show the code.


<form method="post" action="" enctype="multipart/form-data" class="card p-4 demo-form">
        <input type="file" name="file" required class="form-control m-2" />
        <input type="submit" name="importSubmit" value="Import" class="btn btn-primary btn-lg m-2 " />
      </form>


      <?php
      if(isset($_POST['importSubmit'])){
         $csvMimes = array('text/x-comma-separated-values', 'text/comma-separated-values', 'application/octet-stream', 'application/vnd.ms-excel', 'application/x-csv', 'text/x-csv', 'text/csv', 'application/csv', 'application/excel', 'application/vnd.msexcel', 'text/plain');
              if(!empty($_FILES['file']['name']) && in_array($_FILES['file']['type'],$csvMimes) ){
                  if(is_uploaded_file($_FILES['file']['tmp_name'])){
                     //open uploaded csv file with read only mode
                      $file_name = rand(1,9999).time().$_FILES['file']['name'];          
                      move_uploaded_file($_FILES['file']['tmp_name'],"uploads/".$file_name);
                     ///read file and seeds data in table
                      $sql = "LOAD DATA LOCAL INFILE 'uploads/$file_name'
                            INTO TABLE all_fee_data
                            FIELDS TERMINATED BY ','
                            OPTIONALLY ENCLOSED BY '\"' 
                            LINES TERMINATED BY '\n'
                            IGNORE 6 LINES 
                             ";
                          $connection->query($sql);

                  }else{ // end checking if file uploaded or not
                       ?>
                          <div class="alert alert-danger">Problem in uploading file! Plz check  post_max_size and upload_max_filesize in php.ini once.</div>
                       <?php 
                  } 
                  
                } ///empty and mime check end
              } /// isset check end
       
      ?>    



Latest Published Blog


PurgeCSS - Remove unused CSS from Web Pages

PurgeCSS - Remove unused CSS from Web Pages

03-04-2023

It is generally a good practice to remove unused CSS from web pages. As mentioned earlier, removing unused CSS can provide several benefits, including faster page loading times, improved website performance, reduced page weight, and better code maintainability.


Installing Asterisk16 on CentOs7

Installing Asterisk16 on CentOs7

14-04-2022

Here is Step-by-Step Guide to Installing Asterisk16 on CentOs7.


Make CentOS7 Full Screen Like Primary Operating System in VirtualBox

Make CentOS7 Full Screen Like Primary Operating System in VirtualBox

01-03-2022

Learn How to Make CentOS7 Full Screen Like Primary Operating System in VirtualBox. After Installing CentOS in VirtualBox, CentOs Screen does not Occupy the Full Window Screen. Here in this Tutorial, We Will See how to do it.


Setting  Key to a Column For Quick Fetch Operation From Larze Records

Setting Key to a Column For Quick Fetch Operation From Larze Records

14-02-2019

To set a key to a column in SQL, you need to create an index on that column. An index is a data structure that allows the database to quickly look up records based on the values in the indexed column. By creating an index on a column, you can improve the performance of queries that filter, sort, or group by that column.